arXiv:math/0211418vl [math.NT] 26 Nov 2002 


Automorphisms of x^ip) 

Matthew Baker and Yuji Hasegawa 


1. Introduction 


Let Y be a positive integer, and let Xo{N) be the modular curve corresponding to 
the congruence subgroup 


ro(iv) = 


a b 
c d 


G SL2(Z) 


c = 0 mod N 


It is known by |]I| that the full normalizer ^(Tq{N)) of ro(A^) in GL^(Q) = {a € 
GL 2 (Q) I deta > 0} contains the group Tq{N) generated by all the matrices of the 
form 

{%c N'd) ’ ^ ^ 

and that the factor group W{N) := Tq{N)/Tq{N) is an elementary 2-abelian group of 
order where uj{N) is the number of distinct prime divisors of N. In case N is 

square-free, it is also known |14] that rQ(Ai) is maximal as a Fuchsian group (hence 
coincides with iJI(ro(Y))). Now put B{N) = iTI(ro(Y))/ro(Y), which is naturally 
regarded as a subgroup of the automorphism group AutXo(N') of the curve Xq{N). 

Recall that AutA'o(N') has been completely determined lHI[0]0i it agrees with 
B(N) whenever Xo(N) has genus g > 2 and N ^ 37,63. (For these two exceptional 
cases, the group B{N) is of index 2 in Aut Ao(Y).) In particular, if N is square-free, 
then we have Aut Ao(AI) = W{N) whenever Xq{N) has genus g >2 and N ^ 37. 

Let Xq{N) be the quotient curve of Xq{N) by W{N), and let g*{N) be the genus 
of Xq{N). If Y is a prime power, then Xq{N) coincides with Xq{N), the quotient of 
Xq(N) by the Atkin-Lehner involution. We are interested in determining the group 
AutAiQ(Y) (for g*{N) >2). In the rest of this paper, we always assume N to be 
square-free, unless otherwise specified. Then Tq{N) is a maximal Fuchsian group, so 
every nontrivial automorphism of Yq (N) is necessarily exceptional in the sense that it 
does not arise from a linear fractional transformation on the complex upper half plane. 
Thus, if we assume that g*{N) > 2, we expect the group Aut Ag(Y) to be very small. 
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14K02,14H25,11G30,11G10,14H40,14K15. 
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‘Throughout this paper, if X is a curve defined over a field k, we denote by Aut X the group 
Autfc. ( X X k ks) of automorphisms of Xk, over a separable closure ks of k, where Xk, ~ X Xk ks. By 
Lemma 2.2 below, this is the same as Autre(X x.k K) for any separably closed field extension K of k^. 
Similarly, if A is an abelian variety over k, then End A will be shorthand for Endfc„(A Xk ks). 
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The purpose of this note is to prove that when iV is a prime number, this is indeed the 
case: 

Theorem 1.1. Let p he a prime such that g*{p) > 2. Then 


AutXo(p) = 



if 9* (p) = 2; 
if9*{p)>2. 


We remark that Ogg’s method in |Q for determining Aut Ao(p) does not readily 
generalize to Xq{p). Ogg’s proof makes essential use of the action of automorphisms 
on the cuspidal subgroup of Jo{p). However, as Xq{p) has only one cusp, the cuspidal 
group in Jq{p) is trivial, so no information can be gained in this way. (Here Jo{p) and 
Jq(p) denote the Jacobians of Xq{p) and Aq(p), respectively.) 

Our plan for proving Theorem 1.1 is as follows. In Section we investigate the 
structure of AutAiQ(A^) for N square-free. Using this, we show in Section ^ that 
Aut Aq(p) is trivial for almost all primes p. Finally, by considering the reduction of 
Xq{p) modulo various primes, we show in Sections Q and|| that Aut AiQ(p) is trivial for 
the remaining primes such that g*{p) > 2. 


To put Theorem 1.1 in a larger context, we note that little is known in general about 
the problem of determining the rational points on Ag(p). (For a discussion of what is 
currently known about this problem, including the existence of certain “exceptional” 
rational points for p = 191, 311, see S. Galbraith’s paper |^]). On page 145 of his paper 
1^], Mazur writes (adapting his terminology to ours): “What further rational points 
[in addition to the cusp and the rational CM points] does the curve Xq{p) possess? 
This diophantine question (when the genus g*{p) > 0) is extremely interesting, since 
no known method appears to be applicable to it, for any value of p.” We hope that 
Theorem IT will be useful in future investigations of this problem. In any case, if 
it had turned out that there were unexpected automorphisms of Xq(p) for certain p, 
this might have “explained” the existence of certain exceptional rational points such as 
those found by Galbraith. 

Also, we remark that the curve Xq{p^) coincides with the curve which Mazur in 
[p^ j calls Aspiit(p). The rational points on this family of curves are also not known 
in general (see ||^ for some results in this direction). The problem of determining 
the rational points on Aspiit(p) is related to a famous question formulated by Serre 
asking which elliptic curves E/Q, have the image of their mod p Galois representations 
contained in a proper subgroup of GL 2 (Fp) (in this case the normalizer of a split Cartan 
subgroup). See j^j for further details. It would be especially interesting for this reason 
to prove a result similar to Theorem IM in which the automorphism group of Ag(p^) 
is determined. Our methods shed only partial light on this problem. 

The authors would like to thank Brian Conrad and the anonymous referee for their useful 
comments on an earlier version of this manuscript. 


2. Structure of AutA'o(A^) 

In this section, we investigate the structure of AutXQ(A^) for N square-free. It 
is known in this case (see (P) that Jo{N), and hence Jq(A^), is a semistable abelian 
variety over Q. We begin by recalling the following fact: 

Lemma 2.1. Let X he a smooth, proper, and geometrically connected algebraic curve 
of genus g >2 over a field k, and let J = Pic^^^i, he the Picard (Jacobian) variety of X 
over k. Then Autfc(A) injects into EndA;(J). 
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Proof. Suppose cj) G Autfc(X) induces the identity map on J. Then (j) acts trivially on 
the cotangent space at 0 in J, which is canonically isomorphic to It follows 

that (p acts trivially on the image of X under the canonical map to projective space. 
Now if X is not hyper elliptic, then this canonical map is an embedding, which implies 
that (p is identity on A. If X is hyperelliptic, this argument shows that p must be 
either the identity or the hyperelliptic involution h. Since h acts as —1 on J, it follows 
that the natural map from Autfc(X) to Endfc(J) is injective in the hyperelliptic case as 
well. ■ 

We also have the following general fact; 


Lemma 2.2. Let X as above he a smooth, proper, geometrically connected curve of 
genus g > 2 over a field k, and let K/k he an extension field. Let kg denote the 
separable closure of k in K. Then every automorphism of Xk is defined over kg. 

Proof. Grothendieck’s theory of the Hilbert scheme (see 0) implies that there is a 
scheme Aut^/fc, locally of finite type over k, which represents the functor associating 
to each A:-scheme S the set of S'-isomorphisms from X S to itself. It is enough to 
prove that is etale over k, since this implies that every AT-point comes from a 

fcs-point. This is a special case of Lemma 1.11 of [||. ■ 


Corollary 2.3. Let X,k, J be as in Lemma 2.1, and let K/k be an extension field. If 
p is a K-automorphism of Xk := X K such that the induced K-endomorphism p* 
of J X I- K = Pic^^^ is defined over k, then p is defined over k. 

Proof. We may clearly enlarge K so that it is assumed to be separably closed. If kg 
denotes the separable closure of k in K, then by Lemma 2.2, we may assume without 
loss of generality that K = kg. Descent theory now tells us that p is dehned over k if 
and only \i p^ = p for all g G Gal[kg/k). Since the natural map from Autfc^(X kg) 
to Endfc^(J Xfc kg) is injective, it suffices to show that {p^)* = p* on J x^. kg for all g. 
But {p^)* = {p*)^ by the definition of Picard functoriality, so the hypothesis that p* is 
defined over k gives us what we want. 


Lemma |2.1| and Corollary 2.3 are particularly useful when the torsion part of 
End(Pic*^(X))^ has a simple structure. For example, we obtain the following result: 


Proposition 2.4. Let X he an algebraic curve over Q of genus g > 1. Assume that its 
.Jacobian J{X) is semistable over Q and that End J(X) 0 Q is a product of totally real 
fields. Then AutX is elementary 2-ahelian: AutX = Z/2Z0 • • • 0Z/2Z. Moreover, 
every automorphism of X is defined over Q. 

Proof. The first assertion follows from Lemma [2T| , since by assumption the torsion part 
of (End J(X))^ is isomorphic to Z/2Z 0 • • • 0 Z/2Z. The latter part is a consequence 
of Corollary |2.3| and the following result of Ribet. ■ 

Theorem 2.5 (Ribet H^). Let A he a semistable abelian variety over Q. Then every 
endomorphism of A is defined over Q. 


Let Jq{N) be the Jacobian variety of Xq{N). Since N is square-free, Jq{N) is 
semistable over Q (see @). It then follows from Theorem ^ and the results of that 
End Jq (N) 0 Q is isomorphic to a product of totally real fields. Applying Proposition 
we have 


Corollary 2.6. The group AutXQ(A) is elementary 2-abelian. Moreover, every auto¬ 
morphism of Xq (X) is defined over Q. 
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As an application we can determine Aut AQ(iV) when Jq{N) is simple. Recall that 
Xq{N) is hyperelliptic if and only if it is of genus 2 (cf. 0)- Therefore 


Corollary 2.7. If Jq{N) is simple, then 


kntX^{N) 


T.ITL ifg*{N) = 2; 
{1} ifg*{N)>2. 


Now let p be a prime number. Then Xq{p) is hyperelliptic (i.e., g*{p) 
only if 


p = 67, 73,103,107,167,191. 


2) if and 


Since Jq (p) is simple for these six primes (see [^, Table 5]), we conclude that Theorem 
Q holds when g*{p) = 2. 


3. AutXQ(p) IS TRIVIAL FOR ALMOST ALL p 

In the previous section we saw that Aut Ag(A^) is an elementary 2-abelian group. 
The purpose of this section is to show that Aut Aq (p) is trivial for almost all primes p. 

Throughout this section, let p be a prime number and let A be a square-free integer 
such that g*{p) and g*{N) are at least 2. 

We need the following two lemmas. 

Lemma 3.1. Letu be an automorphism o/Aq(A). Then, as endomorphisms of Jq{N), 
we have 

uTi =Tiu 

for each prime I, where Ti is the l-th Hecke operator. 

Proof. This follows from that fact that since N is assumed to be square-free, End Jq {N) 
is commutative (cf. ||l^, Lem. 2.6]). ■ 

Let oo denote the cusp on Xq{N) which is the image of the cusp oo on Xq{N) under 
the natural map. 


Lemma 3.2. If u is a nontrivial automorphism of Xq{N), then uoo ^ oo. 


Proof. (See also |^.) The eigenvalues of u acting on the space of holomorphic differ¬ 
entials of Xq{N) are all ±1. If they are all -|-1, then an argument using the canonical 
map to projective space as in the proof of Lemma 2.1 shows that u = id on Xq(N). 
Similarly, if the eigenvalues are all —1, then X is hyperelliptic and u is the hyperelliptic 
involution. In this case, one can see explicitly using Weierstrass points that uoo ^ oo. 
Recall that Xq{N) is hyperelliptic exactly when g*{N) = 2. The Weierstrass points on 
a genus 2 curve X are the fixed points of the hyperelliptic involution, which is equiva¬ 
lent to saying that P is a Weierstrass point if and only if there is a nonzero holomorphic 
differential vanishing to order 2 at P. In our situation, by computing the (^-expansion 
at oo of each Hecke eigendifferential in the cases where g*{N) = 2, one finds that oo is 
not a Weierstrass point on Xq{N), so it cannot be fixed by the hyperelliptic involution. 

Suppose, then, that u has eigenvalues -|-1 and —1. Then one can choose two dif¬ 
ferentials a;i,a ;2 € II^{Xq{N),Q^) with u*uji = uji,u*uj 2 = —^2 which are normalized 
eigendifferentials for the action of the Hecke algebra T. The g-expansion at oo of each 
tOi is of the form 

7 q 

lit — ± 

with ai = 1. 


UJo — 


E 

m=l 


amq 
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We see that ui = 001+^2 does not vanish at 00 , but its pullback u*u: = uji—u )2 does 
vanish at cx). Consequently, we must have uoo 7 ^ 00 . ■ 

Note that since 00 is the only cusp on Xq{N) when N is square-free, the statement 
uoo 7 ^ 00 is equivalent to the statement that uoo is not a cusp of Xq{N). 

We now come to the following key lemma. 

Lemma 3.3. Let p be a prime number such that g*{p) > 2, and suppose there is 
a nontrivial automorphism u of Xq{p). Let 00 be the eusp of Xq{p), and P a non- 
cuspidal geometric point of Xq (p). Then for each prime I such that I 7 ^ p, the divisor 
Di p := (uTi — Tiu){oo — P) on Xq{p)q is non-zero but linearly equivalent to zero. 

Proof. The divisor Di p is linearly equivalent to zero by Lemma Also, note that 
by Lemma Q := uoo is not a cusp. Now suppose that the divisor Di p is identically 
zero, i.e., that uTioo + TiuP = Tiuoo + uTiP. Since T/oo = {l + l)oo, we see by applying 
u to both sides that there can be no cancellation between points in the support of uTioo 
and uTiP, because all points in the support of TiP are noncuspidal by the definition of 
the correspondence Tp Therefore we must have complete cancellation between uTioo 
and Tiuoo. In other words, we must have {uTi — Tiu){oo) = 0, i.e., (Z + 1)Q = Ti{Q). 
It follows that Q is a rational point of Xq{p) represented by a CM elliptic curve. By 
1^, Thm. 3.2] (and the remarks following the proof of that theorem), we see that Q 
must be a Heegner point of class number one. (Note in particular that all noncuspidal 
rational points on Xq(p) coming from rational points on Xq(p) are Heegner points of 
class number one when g'o(p) > 2, and this happens only for p = 67 and p = 163). Let 
{E,C) represent a geometric point P of Xq{p) which projects to Q, so that Q itself 
can be thought of as corresponding to the set {P, WpP} represented by the unordered 
pair {{E, C), {E/C, E\p]/C)}. Since Q is a Heegner point of class number one, complex 
multiplication theory tells us that E/C = E. If D is a cyclic subgroup of order I of E, 
then E/D is isomorphic either to E or to E/C, since (Z + 1)Q = Ti{Q) as divisors on 
Xq{p). But E/C is isomorphic to E, so we conclude that E/D = E for each of the Z + 1 
cyclic subgroups D d E oi order Z. The resulting degree Z maps from E to E/D = E 
give rise to Z -|- 1 distinct elements of norm Z in End(LZ)/{units}. This is impossible, 
however. To see this, let O = End(E), let K be the fraction field of O (which is an 
imaginary quadratic field), and let R be the ring of integers (maximal order) of AT. If 
the units of O coincide with those of R (which is automatic except when K = Q(i) 
or Q(a;), where i and uj are primitive fourth and third roots of unity, respectively), we 
would get Z -|-1 distinct ideals of norm Z in R, which is clearly impossible since Z + 1 >3. 
We are left with the possibility that R = Z[i] or Z[a;] and O is a nonmaximal order in 
K. However, it is easy to see that these exceptional cases are also impossible using the 
fact that in Z[z], 2 ramifies and 3 is inert, and in Z[a;], 3 ramifies and 2 and 5 are inert. 


Corollary 3.4. Suppose there is a nontrivial automorphism u of Xq{p). Then there is 
a finite morphism f\XQ{p) of degree at most 6, defined over Q. Moreover, every 

nontrivial automorphism of Xq{p) has at most 12 fixed points. 

Proof. We let Z = 2 in the above lemma. Eor the first assertion, take P = uoo. For the 
latter, take P 7 ^ 00 ,uoo and consider a morphism /: Xq{p) —> such that (/) = Di^p. 
Then u*f 7 ^ f, since a comparison of divisors shows that u*f has a zero at 00 whereas 
/ does not. Therefore the assertion follows from the following general fact. ■ 
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Lemma 3.5. Let X be a smooth projective curve over an algebraically closed field k, 
and let u be a nontrivial automorphism of X having r fixed points. If there is a finite 
morphism f:X —> of degree d such that u*f 7 ^ f, then r < 2d. 

Proof. Let us write g = u*f — f. Assume first that g is constant. Then all the fixed 
points of u must be poles of f, so 2d > d > r. Next assume that g is non-constant. 
Then 2d — r' > deg g, where r' is the number of fixed points of u which are poles of 
/. Furthermore, since every fixed point of u which is not a pole of / is a zero of g, we 
must have 2d — r' >r — r'. ■ 

Now let I be a prime such that I 7 ^ p. By abuse of notation, we let Xo{p) denote 
the standard model of Xo{p)q over Z[l/p], and we let Xo{p) be the mod I reduction of 
Xq{p), which is again a smooth curve. 

By Corollary |3.4| there must exist a nonconstant morphism Xq{p) —> of degree 

at most 12 over Q whenever Aut Xq [p) is nontrivial. 

We claim that there necessarily exists a nonconstant morphism of degree at most 
12 from Xq{p) to Pp defined over F;. This follows from the following general lemma 
(c.f. 1]^, Chapter III, Lemma 2.6]): 

Lemma 3.6. Let R be a discrete valuation ring with field of fractions K, residue field 
k, and uniformizing parameter n. Let X be a regular scheme which is proper and fiat 
over Spec(i?) whose generic fiber Xk is a smooth and geometrically connected curve 
over K. Let X]^ denote the (possibly singular) closed fiber ofX. Then any line bundle 
of degree d on Xk extends to a line bundle C on X, and the pullback Ck to X^ is 
a degree d line bundle with d\m.K H^{Xk.,T-k) < dim^, T^.). 

Proof. With our hypotheses, it follows from [T^, II, 6.11] that there is a natural isomor¬ 
phism between the divisor class group of X and the group Pic(X). The surjectivity of 
the natural map Pic(X) ^ Pic(X/i') then follows from ]p!^ II, 6.5(a)]. (If X/ii is smooth 
then this map is also injective). Let T be a line bundle on X extending the degree d line 
bundle Ck on Xk. If we define the degree of a line bundle C to be x(^) “ then 

this degree is constant on fibers because X/R is flat; by Riemann-Roch this degree 
must be d. 

We have dimj^ H^{Xk, Ck) = dim^^ H^{X, C)®rK because cohomology commutes 
with the flat base extension R ^ K hy ]|T^, III, 9.3]. 

The pullback (restriction) Ck of T is a line bundle of degree d on Xk, and we have 
dimKH^{X,C) ®rK < dimkH^{Xk,Ck) by the semicontinuity theorem (c.f. |^, III, 
12 . 8 ]). 


Corollary 3.7. Let X be as in the above lemma, and assume furthermore that the 
closed fiber Xk is smooth. Suppose Xk admits a morphism of degree at most d to P^ 
defined over K. Then Xk admits a morphism of degree at most d to defined over k. 

Proof. This follows directly from the lemma, since if Y is any smooth, proper, geomet¬ 
rically connected curve over a field k, then there exists a morphism of degree at most 
d from y to P^ defined over k iff there exists a line bundle C of degree at most d on Y 
such that dimfc H^iY, C) >2. ■ 

In our situation, since flpi (Fp) = -|-1, we see that the number ttXo(p)(F; 2 ) cannot 

exceed 12(/^ -|- 1). On the other hand, Ogg [^] gives the following estimate: 

t)Xo(p)(F;2)>^(p+l) + 2. 
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Therefore, if Aut Xq (p) is nontrivial, we must have the inequality 

12{f + l)>^-^{p + l) + 2. 

Setting I = 2, we obtain the following: 


Corollary 3.8. If p > 695, then Aut Aq(p) is trivial. 


The next result follows from Corollary 3.4: 


Corollary 3.9. If Jq{p) has a Q-simple factor of dimension larger than {g*{p) + 5)/2, 
then Aut Ag (p) is trivial. 


Proof. Suppose tt is a nontrivial automorphism of Ag(p), which we know must have 
order 2. By Corollary 3.4, the number r of fixed points of u is at most 12. Let X' 
be the quotient of Xq{p) by u, let g' be its genus, and let J' be the Jacobian of X'. 
Applying the Riemann-Hurwitz formula to the degree 2 map Xq{p) —> X', we hnd that 
2^' — 1 < g*{p) < 2g' + 5, so that 

( 1 ) {g*{p)-if)/2 < g'<{g*{p) +l)/2. 


In particular, if m > {g*{p) + l )/2 and if Jq{p) has a Q-simple factor A of dimension 
equal to m, then g*{p) = dim Jq(p) > m + dim J' > m + {g* {p) — f)/2, which implies 
that m < {g*{p) + 5)/2 as desired. ■ 


Remark 3.10. The Q-simple factors of Jq{N) are the same as the Q-simple factors of 
Jq{N) whenever N is square-free, since in this case Jq{N) is semistable over Q (c.f. [^|; 
see also Theorem I 


Remark 3.11. Let A be a positive integer. We may compute the Q-simple splitting of 
Jq(A) by factoring characteristic polynomials of Hecke operators acting on the space 
S'! {N) of cuspforms of weight 2 on rQ(A). For instance, suppose we are given an element 
T of the Hecke ring Z[{ri};| 7 v] whose action on 5'|(A) has sguare-free characteristic 
polynomial Tr(x). Let T'r = • 'I '2 • • • 'Ls be the factorization of Ty over Q. Then 

deg Ti -b deg 'L 2 H-b deg 


gives the Q-simple splitting of Jq(A). 

Now consider the case g*{p) > 2. (For a discussion of how to compute g*{p), see 
p^ , p. 20]). Using the above observation, we find that there are 39 primes in the range 
p < 695 for which Jq{p) is not simple (see Table |^). Applying Corollary 3.£ to these 
cases, we see that Aut Aq(p) is trivial except possibly for 

p = 163,193,197, 211, 223, 227, 229, 269,331, 347, 

359,383,389,431,461, 563, 571, 607. 


4. Reduction of Xq{p) modulo I 


In this section we consider the reduction of Ag (p) and its reduction modulo I, where 
I is a prime such that I 7 ^ p. Let 5^ (p) be the space of cuspforms of weight 2 on Fg (p). 
For each Hecke-stable subspace S of S^^p), define 




l+l'^ -tvTln\S + 


I ■ trTp- 2 |S' if n > 2 ; 

0 otherwise. 


Formulas for computing the traces of Hecke operators are given in , 13C ]. 
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Table 1. Simple splitting of Jq{p) 


p 

Splitting 

163 

1 + 5 

193 

2 + 5 

197 

1 + 5 

211 

3 + 3 

223 

2 + 4 

227 

2 + 3 

229 

1 + 6 

269 

1 + 5 

277 

1 + 9 

331 

1 + 3 + 7 


P 

Splitting 

587 

5 + 13 

593 

1 + 18 

599 

2 + 11 

607 

5 + 7 + 7 

613 

5 + 18 

647 

2 + 14 

653 

7 + 17 

659 

1 + 16 

677 

1 + 2 + 18 


P 

Splitting 

461 

2 + 3 + 7 

467 

1 + 12 

491 

2 + 10 

499 

2 + 16 

503 

1 + 10 

523 

2 + 15 

547 

2 + 18 

557 

1 + 18 

563 

3 + 3 + 9 

571 

3 + 6 + 10 


P 

Splitting 

347 

1 + 2 + 7 

359 

1 + 1 + 4 

373 

1 + 12 

383 

2 + 6 

389 

2 + 3 + 6 

397 

2 + 13 

431 

1 + 3 + 4 

439 

2 + 9 

443 

1 + 1 + 12 

457 

2 + 15 


We note that since the Q-simple factors of Jo(p) (and hence of Jq{p)) are Hecke- 
stable and appear with multiplicity one (see |^, Chapter II, Section 10]), it follows that 
if Y is any curve of positive genus covered by Xg(p), then the image of the Jacobian 
of Y in Jq (p) via Picard functoriality is Hecke-stable. This implies that the subspace 
S of S^ip) (which is canonically identified with the cotangent space of Jq{p) at the 
origin) corresponding to the cotangent space of J at the origin is Hecke-stable. We say 
colloquially in this situation that the Jacobian of Y “comes from” S. 

Before giving the next proposition, we recall the following fact, which is proved, for 
example, in [^]. Suppose ii is a discrete valuation ring with fraction field K, and that 
f : X ^ Y is a finite morphism of projective, smooth, geometrically connected curves 
over K. Assume furthermore that X{K) is nonempty. Then if X has good reduction, 

Y does as well. (We say a curve X as above has good reduction if there exists a scheme 
of finite type which is proper and smooth over R and whose generic fiber is X.) 

Proposition 4.1. Suppose there is a nontrivial automorphism u of Xq{p). (By the 
remarks at the end of section 2, we may assume that g*{p) > 2, so that Xq{p) is not 
hyperelliptic.) Let Y = Xq{p)/{u) be the quotient of Xq{p) by the action of u, which is 
a curve of genus g >1, and let Y be the reduction ofY modulo 1. If the Jacobian J of 

Y comes from S, we must have tlT(F;n) = 

Proof. We can write tlT(F;n) = 1 + P — + Ui"'); where the (a*,a*) are 

conjugate pairs of eigenvalues of Frobenius acting on the T-adic Tate module of J for 
any prime I' ^ 1. On the other hand, since u is defined over Q, the Jacobian of Y may 
be considered, up to Q-isogeny, as an abelian subvariety over Q of Jq{p)- Thus from 
the Eichler-Shimura congruence relation we conclude that the set {ai + ai}f=i gives the 
eigenvalues of Ti\S. Using the standard recurrence relations for the Hecke eigenvalues, 
we obtain the desired equality. ■ 

It follows from this proposition that S cannot correspond to the Jacobian of Y 
once we know uin := tlXQ(p)(F;n) > 2 • for some prime power P. Checking 
this inequality for all S / {1} such that {g*{p) — 5)/2 < gs < {g*ip) + l)/2 (see 
(® above), where gs is the dimension of S, we see that Aut A'g(p) is trivial for p = 
163,193,197, 269, 277,331, 347, 359,383,461, 563,607 (Table |). 
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Table 2. 


p 

r 

I2ln 


ffs 

163 

2 

8 

3 

1 

193 

5 

14 

6 

2 

197 

3 

12 

4 

1 

269 

3 

9 

4 

1 

331 

4 

20 

5 

3 


P 

r 



96 

331 

4 

20 

8 

4 

347 

3 

11 

4 

3 

359 

4 

17 

8 

1 

359 

4 

17 

8 

1 

359 

5 

10 

4 

2 


P 

r 

I/p 


9S 

383 

9 

38 

15 

2 

461 

9 

42 

19 

5 

563 

4 

31 

15 

6 

607 

4 

27 

11 

7 

607 

4 

27 

12 

7 


It remains to determine Ant Xq{p) for the seven primes 
p = 211,223,227,229,389,431,571. 


5. Reduction of Aq(p) modulo p 


In this section, we consider the reduction of Xq (p) modulo p. 

Let Xq{p) be the Deligne-Rapoport model of Xq{p) over Z(p), whose special fiber 

Xo{p) can be identified with a union of two copies of Ao(1)fp — Pp intersecting trans¬ 
versely at the supersingular geometric points. The automorphism Wp of Xq{p) induces 
an automorphism Wp of the special fiber Xq{p) which interchanges the components Yi 

and Y2 of Xq{p). It takes each supersingular geometric point to its Fp 2 -conjugate, and 
in particular the fixed points of Wp correspond precisely to the supersingular j-values 
in characteristic p which are defined over Fp (See |24| and |Q). 


The quotient of Xq (p) by Wp is an arithmetic surface Xq (p) whose generic fiber is the 

smooth curve Xq{p), and whose closed fiber Xq{p) is a copy of the j-line that intersects 
itself transversely at the g* = g*{p) geometric points corresponding to conjugate pairs 
of supersingular elliptic curves defined over Fp 2 but not Fp. The fact that the closed 

fiber of Xq (p) = Xq (p) / Wp is the quotient of Xq (p) by the action of Wp follows from the 
discussion in |l^. Appendix A7], especially Prop. A7.1.3.[] 

One can see that the fixed points of Wp on Xq{p) get mapped to smooth points of 

Xq (p) by using the fact that the components Ti and Y2 of Xq (p) intersect transversely, 
plus the following local calculation: if k is any field, the subring of k[[x,y]]/{xy) con¬ 
sisting of all elements left invariant by the automorphism which interchanges x and y 
is isomorphic to k[[t]], a power series ring in one variable. 


We assume for the rest of this section that g*{p) > 0. 


Note that if p ^ 1 mod 12, then Xq{p) is not regular: the supersingular points 
corresponding to elliptic curves with extra automorphisms, i.e., to elliptic curves with 
j-invariant 0 or 1728, are not regular. However, the quotient of Xq{p) by Wp behaves 
better in this respect: 


^Specifically, we are using the fact that (a) the order 2 of the group generated by Wp is invertible in 
Fp since p ^ 2, and (b) the order of all stabilizer groups of all geometric points of Xo(p) are invertible 
in Fp since p yf 2, 3. The reason we need (b) is that the Deligne-Rapoport model is only a coarse 
moduli scheme, but one can show under the hypothesis in (b) that formation of the coarse moduli 
scheme associated to an algebraic stack commutes with the base change in question. 
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Lemma 5.1. The arithmetic surface Xq{p) is the minimal proper regular model over 
Z(p) for X^{p). 


Proof. We first claim that Xq(p) is indeed regular. Note that Xq(p) is normal and its 
closed fiber is generically smooth. Since the generic fiber Xq{p) is smooth, the only 
points which are potentially not regular are the finitely many singular points on the 


closed fiber. Let s G XQ{p)(fFp 2 ) be such a point, corresponding to a supersingular 
j-invariant j{s) defined over Fp 2 but not Fp. In particular, j{s) is not 0 or 1728, and 
therefore 7r“^(s) consists of two distinct regular points si,S 2 of Xq(jp), where vr is the 
canonical map from Xq{p) to Xq{p). Since si and S 2 are not fixed points of Wp, it 
follows from []T6| , Thm. A7.1.1] that vr is etale over s. Therefore s, being the image of a 
regular point by an etale map, is regular. 

To see that Xq{p) is minimal, it suffices by Castelnuovo’s criterion ([Q, Thm. 3.1]) 
to note that Xq{p) has no exceptional divisors in the sense of |Q, Def. 1.3]. This follows 
immediately from the fact that the special fiber is irreducible, and therefore by ]Q, 
Cor. 4.1] has self-intersection zero. ■ 

Let u be an automorphism of Xq{p). By the defining property of the fact that Xq{p) 
is a minimal proper regular model (see ]Q, Def. 1.1]), u extends to an automorphism of 


the scheme Xq{p). It therefore induces an automorphism u of the special fiber Xq{p) 
and an automorphism a of the normalization Pp of X^{p) which permutes the set of 
2g* supersingular geometric points defined over Fp 2 but not Fp. 

We claim that if u is not the identity map on the generic fiber Xq{p), then u is not 
the identity on the special fiber. 

This follows from Lemma 2A and the fact that Jq (p) has semistable reduction at p, 
together with the following two basic results: 


Theorem 5.2 (Raynaud). Let R be a discrete valuation ring with fraction field K. Let 
X/R be a proper, flat, and regular curve whose generic fiber is smooth and geometrically 
irreducible and whose closed fiber is geometrically reduced. Let X he the generic fiber of 
X, and let J be the Neron model of the Jacobian of X. Then the connected component 
of the identity in J coincides naturally with the relative Picard scheme Pic^jj^. 
In particular, every automorphism tp of X induces an endomorphism of , and if fj 
is the identity map on the special fiber of X then it induces the identity map on the 
connected component of the closed fiber of J. 

Proof. This is a special case of Theorem 4, Section 9.5]. ■ 


Lemma 5.3. If G is a semiabelian group scheme over a discrete valuation ring R, then 
an endomorphism of G inducing the identity map on the special fiber is necessarily the 
identity. 

Proof. We may assume without loss of generality that R is complete, and we denote 
by m its maximal ideal. 

Let u € End G be such that Ug (the induced map on the closed fiber G^ of G) is the 
identity map. We want to show that u = 1 on all of G. It is more convenient to work 
with the endomorphism v := u — 1 which satisfies Vs = 0. We want to show that u = 0. 

Fix an auxiliary prime I not equal to the residue characteristic of R, and let Hn = 
G[P] for each integer n > 1. Since G is semiabelian, it follows from |^, Lemma 2, 
Section 7.3] that Hn is a quasi-finite, flat, etale, and separated group scheme over R for 
all re. Since R is complete, and therefore henselian, we may consider, for each re, the 
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finite part {HnY of i/„, which is a finite etale open and closed subscheme of having 
the same closed fiber as Hn (see Section 7.3, p. 179]). As = 0 on the closed fiber 
of {HnY, and {HnY/R is finite etale, it follows that u = 0 on {HnY. 

In particular, u = 0 on all infinitesimal closed fibers of {HnY (i.e., on {HnY 
for all integers k > 1). We claim that u = 0 on all infinitesimal closed fibers of G. To see 
this, note that the infinitesimal closed fibers of {HnY of are the same by the con¬ 
struction of {HnY ■ Also, since Gg is semiabelian, the collection of subschemes {(77„)s} 
is topologically dense in Gg. It follows by [ p!0| . Proposition 11.10.1,Theorem 11.10.9] 
that the collection of subschemes induced by {{Hn)} is schematically dense in each 
infinitesimal closed fiber of G. This proves our claim that u = 0 on each infinitesimal 
closed fiber of G/R. 

It now follows that v induces the zero map on the formal completion of G along the 
identity section. Therefore u = 0 on the generic fiber of G, and since G is flat over R, 
it follows that u = 0 on G as desired. ■ 

Denote by Y the normalization of Tq(p), which as we have seen is isomorphic to 
Pp^, and let {ei,e{} be the conjugate pairs of j-values of supersingular elliptic curves 

which are identified in the map Y —> Tg (p) {i = 1,..., g*). 

We conclude from our above discussion that if there is a nontrivial element u € 
AutAQ(p), then there is an element a G Aut(Pp^) = PGL(2,Fp) of order 2 which 
permutes the set {ei,e{} of 2g* supersingular j-invariants which are not Fp-rational. 
Now if p > 3 is an integer, then there is no nontrivial automorphism of P^ permuting 
a general set of 2g points. So if g* > 3, it is unlikely that an automorphism a as above 
exists; in any case, for each specihc value of p one can compute whether or not such an 
a exists, since the set {ei,e{} can be computed explicitly (see Table ^). 

We therefore check by an explicit computation that there is no such element a G 
PGL(2, Fp) for the remaining seven cases p = 211, 223, 227, 229, 389,431, 571. It follows 
that Aut Xq(p) = {1} for these values of p. This concludes our proof of Theorem 1.1. 
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